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DESCRIPTION 

DliGITAL REMOTE CONTROL DEVICE 



5 This invention relates to a remote control for a multimedia TV receiver. In 

particular the invention relates to a remote control with a display and two-way 
communication with a multimedia TV receiver that is configured to enable a 
variety of novel, interactive applications. 

lo GB-A-2343073 describes a remote control with a display which is 

configured for the display of electronic programme guides (EPG), teletext and 
internet content remotely from a TV set. The document further describes a facility 
for the instant replay of part of a televised program being watched by a user. 
WO 00/59212 describes a remote control also capable of providing on a 

15 display remote from the TV. display, ERG and other information and data. In 
addition, the remote control described in WO 00/59212 is provided with a soft 
graphical user interface (GUI) and speech recognition facility whereby the user ' 
can, for example request information from the ERG related to a specific topic 
(such as sport, movies or a named TV show). 

20 The described prior art remote controllers are limited in function in that 

they provide only information or data specifically requested by the user. Also, 
video images, which are displayable remotely using these prior art devices, are 
restricted by the channel selection being viewed on the TV at the time of use of 
the remote control. 

25 

The present Invention aims to further extend the user applications for a 
remote control with a visual display. 

In accordance with a first aspect the present invention provides a remote 
control device for use with a television receiving apparatus, the television 
30 receiving apparatus comprising; a system controller, a receiver for receiving 
broadcast video data and supplemental textual data and a TV display for 
displaying the data, the remote control being operable to communicate with the 
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system controller of the television receiving apparatus and having if s own display 
and user-operable control, wherein, the .remote control device comprises means 
responsive to the user operable-control for requesting and downloading video 
and/or supplemental textual data received by the television receiving apparatus 

5 and not currently displayed on the TV display, to be viewed by the user on the 
display of the remote control device. 

The invention is conveniently implemented where the television receiving 
apparatus system controller comprises two tuners, a first of which is operable to 
find data and information for display on the TV display and a second which is 

10 operable to find data and information for display on the display of the remote 
control device. 

In another aspect, the invention provides a television receiving apparatus 
comprising a receiver for receiving broadcast video data and supplemental 
textual data, a TV display for displaying the data and a system controller, the 

15 system controller comprising a plurality of tuners, at least one tuner being 
controllable by means of a user-operable control associated with a remote control 
device in accordance with the first aspect, to display data on the display of the 
remote control device. 

In a further aspect the invention provides a television receiving apparatus 

20 comprising a receiver for receiving broadcast video data and supplemental 
textual data in a plurality of data streams, a TV display for displaying data and a 
system controller, the system controller having means for receiving and 
processing a request from an alternative display device to display data 
associated with a data stream, other than that (or those) displayed on the TV 

25 display, on the alternative display device. 

In another aspect the invention provides a method for displaying images 
and/or information associated with a data stream received by a television 
receiving apparatus on a display device remote from the television receiving 
. apparatus comprising; in response to a request, locating a data stream having 

30 associated therewith the requested images and/or information; 

extracting the located data stream; defining the extracted data stream in a form 
mutually comprehensible to the television receiving apparatus and the remote 
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display device; transmitting tlie defined data stream to the remote display device; 
and, rendering tlie images and/or infomiation associated with the defined data 
stream on a display screen of remote display device. 

In yet another aspect, the invention provides A method for providing 

5 interactive participation in a TV programme comprising; in a first data stream, 
broadcasting video of the programme to be displayed on a first display apparatus 
associated with a television receiving apparatus; in a second data stream 
providing infomnation relating to the content of the first data stream to be 
displayed on a second display apparatus, which second display apparatus is 

10 provided with a user interface; and, requesting a user response to the infomnation 
contained in the second data stream. 

The invention permits a user to view on the remote control device images 
other than the one being watched on the main TV display. This allows the user to 
follow a program on another channel (for example, to see if an advertisement 

15 break has finished, or a news story of interest has come up) without interfering 
with the viewing of the programme currently on the main TV display. Alternatively, 
the user may view a teletext page associated with a channel other than that 
currently displayed on the TV display, or may access intemet web sites or send 
and receive e-mails all without disrupting viewing on the main TV display. 

2o Conveniently, the invention may be enabled by means of an all-software decode 
mnning on the central processor (CPU) of the set top box (STB) or other system 
controller of an interactive TV receiver. This allows reduced (but satisfactory) 
resolution previews to be displayed on the remote display thereby allowing its 
implementation via relatively simple STB technology with almost zero additional 

25 cost to the manufacturer. 

Desirably, the remote control device is configured to display requested 
data whether or not the TV display is active. When the TV display is active, the 
remote control device will conveniently display images from a TV channel other 
than that being viewed on the TV display. In addition or in the altemative, the 

30 remote control may be configured to request, receive and display images recently 
displayed on the TV display, such as play back of a previous scene in a 
programme currently being viewed on the TV display. This feature may be 
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enabled by interaction between the user operable control and a memory in the 
system controller or by means of a local memory in the remote control device. 

The user operable control may be provided in any convenient form 
including, but not strictly limited to; a keyboard (optionally Incorporating a mouse, 
5 tracker ball or similar feature to assist in selecting from on screen menu displays); 
touch screen technology and/or speech recognition. 

The remote control device may communicate with the system controller of 
the TV receiving apparatus by any suitable means. Desirably, communication is 
via a cordless connection such as (but not strictly limited to) infrared. 

10 Desirably, the remote control device is configured to interact with other 

audio/visual apparatus such as a DVD player, VCR or games console. In such 
embodiments, the remote control is provided with means responsive to the user 
operable control for requesting, receiving and displaying data or information from 
the alternative audio/visual device other than is currently being displayed on the 

15 TV display. Such features permit a user, for example, to view a programme being 
recorded on a VCR whilst an alternative programme is viewed on the TV display. 
Alternatively, the user may remotely view a TV programme cun^ently being 
broadcast, whilst a movie recorded on a DVD is being played on the TV display. 
It is known for digital TV broadcasts to comprise both video data and 

20 textual information which may comprise separate data streams. The data streams 
may be transmitted in tandem in a single carrier signal. Such arrangements can 
be used to provide interactive data such as the positions of contenders in a race, 
newsflashes, or the latest score in a sports match. It is envisaged that such 
technology could be used to provide an interactive medium via the display of the 

25 remote control device of the invention. For example, the user operable control 
may be configured to download quiz questions to the remote control display and 
to upload answers to these questions. Again, the user may have this interactive 
experience without interfering with the enjoyment of a TV programme by other 
viewers of the main TV display. 

30 In the case of a suitably live broadcast sporting event, position data of 

players/participants may transmitted by a broadcaster in tandem with a live 
broadcast (for example, by image processing on a video feed from a fixed 
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camera). In addition, a graphic representation of tlie piayers may be 
superimposed on a representation of the course/pitch/tracit at which the live 
sporting event is occurring and downloaded for display on the screen of the 
remote control device whilst the live action is broadcast on the main TV display. A 
continuously updated field displaying event statistics may augment the graphic 
representation. By interaction with the remote control device by means of the 
user-operable display, the user may choose control options for the display of the 
event on the TV display. For example, the user may select camera angles, audio 
feed locations and the like simply by indicating desired locations on the graphic 
representation. The user selections may then be uploaded to the broadcaster via 
the system controller of the TV receiving apparatus and the camera view/audio 
feed may be transmitted to the remote and/or TV display in accordance with the 
user's requirement. 

In addition, or in an alternative, where the live action video and graphic 
representation are stored in conjunction, the graphic representation (stepped 
through at high speed) would allow a user to quickly identify interesting passages 
of play and then mark them to create a highlights package from the recorded 
video. 

In a further development of the aforementioned interactive applications, it 
is envisaged a broadcasting company could encourage channel loyalty through a 
"loyalty points" scheme. The "points" are provided in the fonn of electronic credits. 
Such schemes are well known in the retail industry where purchasers are 
awarded points electronically via a loyalty card in return for spending a certain 
amount of money in a store. In the proposed scheme of the invention, 
broadcasters send points (carried in an electromagnetic signal) to a TV receiver 
embedded in a video broadcast. The remote control device is provided with a 
user operable control for requesting and downloading these loyalty points and a 
memory for storing the points. In further broadcasts, the broadcaster may send 
interactive information directly to the remote control device via the same route as 
the points. Such interactive information may provide the user with opportunities to 
spend loyalty points or to acquire additional points, for example by betting points 
on the outcome of a future event posed by the broadcast interactive infomiation. 
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Points accrued may be used to purchase items or services offered for sale 
by the originating broadcaster. For exampie, points may be traded for viewing of 
a pay-per-view presentation. Alternatively, character merchandise associated with 
a flagship TV show broadcast by the broadcaster may be purchased with the 

5 loyalty points in place of actual currency. In a further alternative, telephone calls 
or Internet access made utilising the same sen/ice provider as supplies the 
interactive TV data may be paid using accmed points. Points may be awarded by 
the broadcaster as a reward for certain behaviours, for example watching 
advertisements or buying goods or sen^ices. 

10 A particular embodiment provides a gaming/gambling arrangement based 

on the "loyalty points" system, and having a number of particular features: points 
may be given freely by a broadcaster to encourage the user to gamble with them. 

The gaming/gambling activity is in the form of questions appearing on the 
remote display, relating to a TV programme which the user is currently watching 

15 on the main TV display. The remote control device is suitably configured such 
that the broadcaster can "wake up" the device from a dormant state to present 
questions to the user; and user response (or lack of it) to gaming questions may 
be used to build up a user profile. User profiles may subsequently be used by the 
broadcaster for selective advertising and marketing purposes. For example, 

20 where a user is recognised to respond to questions related to sporting 
presentations, the broadcaster may target the user with advertisements directed 
to sporting magazines, sports clothes and future, pay-per-view sporting 
presentations to be broadcast by the broadcaster. 

Other possibilities will no doubt occur to the skilled artisan without 

25 departing from the true scope of the invention. 

In another aspect, the remote control device of the invention may be used 
in association with digital recording apparatus, for example a video 
camera/recorder. Scenes recorded by the camera/recorder could be remotely 
downloaded and edited using the remote display and an appropriately configured 

30 user interface. The complete edited version could then be uploaded back to the 
camera and recorded on a data carrier for reproduction and personal use. 

In a further interactive application, a broadcaster may have the facility to 



wo 03/085965 



PCT/IB03/01194 



7 

'•wake" the remote control device from a dormant state, for example by 
downloading an activation code to the device or the system controller. Once the 
device has been awakened, the broadcaster may selectively send data and 
information to the remote control device for display. Information may be provided, 

5 for example, in the form of downloads of virtual leaflets containing advertising 
data related to material that the user is currently viewing on the TV display. 
Where the programme being viewed is a commercial, the data may comprise 
background infomiation or details of other items in a similar product range. If the 
user chooses to ignore the downloaded leaflet, it is removed automatically from 

10 the remote control device after a defined period, - however, if the user 
acknowledges receipt of the leaflet within the defined period, by interaction with 
the remote control device, details on the subject matter of the leaflet and the 
nature of the user's response is passed back to the broadcaster who may use the 
data to build up a personal profile of the viewer. The profile may be used to 

15 selectively target individuals with advertisements for goods or sen/ices in which 
they have historically shown an interest. Such advertising may include the 
promotion of TV broadcasts on other channels (provided by the same 
broadcaster) which may be of interest to the user. Leaflets may be sent 
repeatedly and, where the local system permits, they may be locally cached to 

20 avoid the download of such large volumes of data. 

Embodiments of the invention and various aspects thereof will now be 
described. For the purposes of this description, the following terms are to be 
interpreted as defined. 

"Broadband" A transmission medium capable of supporting a wide range 

25 of frequencies. In the context of the described invention, this refers to a high- 
speed (typically 51 2 kbit/s but up to 40 Mbit/s is possible) Internet connection via 
cable, DSL, satelite or wireless local loop. 

"Gatewa/' A device that contains a broadband modem, a wireless 
interface and the associated software to make it an intelligent hub. This may be 

30 a set top box (STB), a personal computer (PC) or a stand-alone device. 

The following acronyms are believed to be commonly used and 
understood in the technical field of the invention: 
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ADSL Asymmetric Digital Subscriber Line 

DCT Discrete Cosine Transformation 

DSL Digital Subscriber Line 

DVB Digital Video Broadcast 

EPG Electronic Program Guide 

FLTK Fast Light Tool Kit 

GUI Graphical User Interface 

IDCT Inverse Discrete Cosine Transformation 

MPEG Motion Picture Experts Group 

PDA Personal Digital Assistant 

PES Packetised Elementary Stream 

POTS Plain Old Telephone System 

PSI Program Specific Information 

SI Service Information 

STB Set-Top Box 

VLC Variable Length Coding 

XML Extensible Mark-up Language 

Desirably, the invention is in the forni of a communicating portable touch 
screen display device that allows Information normally displayed on the TV 
screen to be displayed 'off-screen' and on the display of the remote, touch screen 
device. Information may be displayed on the device that is not currently 
displayed on the TV screen, for example a preview image of another channel or a 
web page associated with a program being watched. This remote display 
capability allows a user to find out whatever information he/she likes, such as 
browsing EPG listings or viewing teletext pages on whatever channel, without 
disturbing the viewing of persons watching the main TV display. Also, when 
appropriately configured, the device enables the user to experience a wide range 
of interactive activities. 

The content displayed by the remote control device of the invention is 
accessed through a gateway device such as a STB, and the device has no direct 
external network connection of its own. It is envisaged that a plurality of such 
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devices may be used simultaneously, yet independently, in relation to a single TV 
receiver. For example, each member in a family may have their own personal 
device. In an alternative, a single device may be equipped with a memory facility 
whereby a plurality of individual user set ups may be stored and selected by 
individual users. 

Whilst the description of the invention focuses on the device as a TV 
companion, it should be appreciated that the device has potential application as a 
universal front end for digital consumer devices such as PVRs, DVDs and MPS 
jukeboxes. As these digital consumer devices offer more and more functionality, 
their user interfaces will become increasingly complex and the device of the 
invention is a highly appropriate platform to present such user interfaces. 

There are two principal features of the device; firstly, the ability to mine 
' infomnation in the gateway device, and secondly, the facility to display images or 
data on the remote display which relate to different transmissions than that or 
those displayed on the main TV display. 

For the purposes of exemplification, an embodiment of the invention will 
now be further described with reference to the Figures in which: 

Figure 1 illustrates the system architecture for an embodiment of a remote 
display device in accordance with the invention, and 

Figure 2 illustrates the software architecture for an embodiment of a 
remote display device in accordance with the invention, 

Rgure 1 illustrates how content anives at the remote control device when it 
is used as a digital TV companion device: 

In Figure 1 , a television receiving apparatus 1 is situated in a user's home 
environment, H. The television receiving apparatus 1 comprises a system 
controller embodied in a STB 2, and a display, 3. In wireless communication with 
the apparatus 1 and also in the home environment H is a remote control device 4 
as previously described. In a location remote from H, is located the service 
provider (for example a TV broadcaster) SP, who transmits his own generated 
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content 5 to receiving apparatus 1 in environment H. In addition, sen/ice provider 
SP may also receive, for example, Internet data 6 and content from partner 
websites 7. The content 6 and 7 can be requested by the television receiving 
apparatus 1 and transmitted via the service provider SP to the STB 2. As 

5 previously described, the remote control device 4 can in turn instruct the STB 2 to 
request certain content 5, 6, 7 from the service provider SP for display on the 
display 8 of the remote control device 4. 

As shown in the Figure, the link between the sen/ice provider SP and the 
home (or location of the user) H may be any of a satellite, cable, terrestrial 

lo broadcast or a telecommunications operator or any other conduit for providing 
digital TV data. The link is composed of several multiplexes, each of which may 
cany several video, audio and/or data streams. The skilled addressee will be 
aware of various methods by which the various streams are multiplexed together 
as these have been previously described in relation to DVB technology. Several 

15 different types of data streams are specified, most notably SI (such as teletext 
EPG or Internet) and PSI (the video/audio broadcast). 

As is also known, certain DVB specifications allow any type of digital 
information to be transmitted as a private data stream. This data stream may be 
sent to all receivers, individual receivers or selected groups of receivers and by 

20 means of suitable software, the device of the Invention can be configured to 
receive this data independently of the data for display on the TV screen. The 
WAN connection shown in Figure 1 is optional, it allows downstream internet data 
to be transmitted to the device of the invention (the upstream connection is not 
shown in the Figure, but could, for example, be a POTS or ADSL modem, a cable 

25 modem or even a satellite back channel). This arrangement facilitates accessing 
of both Internet images and TV video broadcasts simultaneously (an activity 
sometimes referred to as "telewebbing"). Enhanced sen^ice (EPG, teletext and 
the like) delivery is also made possible by this arrangement. 

The following describes the method by which infomnation is caught and 

30 displayed by the device of the invention. 

The display of Information on an external device involves a number of 
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Stages. The information must first be discovered, then extracted, rendered, 
transmitted and displayed. 

Discovery : When an item of information (for example, an EPG description 
or an l-frame for a preview image) is requested, the first step is to detemiine 
5 where that information resides. It may be part of an already existing (for example 
in the STB memory) SI table or it may exist in a different transport stream than 
the one currently being examined. The discovery phase resolves the location of 
the item of information being requested. 

Extraction : Once the location of the information item has been determined 

10 the appropriate information is extracted. The mechanism for extraction will 
depend on the relevant location. If the item was not available In either the STB 
memory or the data stream for the channel currently selected for the TV display, 
then the tuner is instructed to change to the correct frequency (as determined 
during the discovery phase) for receiving the requested information. The PID and 

15 section filters may be configured to build a table that may be examined for the 
requested information item. 

Renderino: With traditional on-screen display of information the rendering 
phase would consist of drawing the appropriate images on the graphics planes 
which are superimposed on the TV picture. The inventive device provides for the 

20 rendering of images on a display other than the TV screen (i.e. the remote 
display). The rendering stage therefore, consists of describing the item of 
infonnation in a form mutually comprehensible to both the gateway device and 
the remote control device of the invention. In the described embodiment, this is 
achieved using XML and the appropriate DTD/schema. 

25 Transmission : The XML description of the requested information is 

transmitted wirelessly to the remote device. This can be achieved by using any 
number of existing wireless protocols known in the technical field of the invention. 
For example, Bluetooth, IEEE 802.1 1 or HomeRF. 

DisDiav : Once the XML description of the requested information item has 

30 been successfully received it is rendered on the display of the remote control 
device. 
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The following describes the functionality of an embodiment of the invention 
for some user applications. Examples of suitable hardware and software 
configurations for an embodiment of the invention are now outlined. 

Two suitable platfomris are the Intel StrongARM SA1 1 1x reference board 

5 and the Compaq iPAQ H3630 PDA. The iPAQ device features very similar 
hardware (in particular, the same CPU) to the StrongARM reference board and 
provides a subset of the StrongARM functionality in a more compact form factor. 

In the embodiment described, the STB is a low-end box based on the 
SGS-Thomson STi5512 chip. This chip features a 60 MHZ ST20 32-bit 

10 processor that provides sufficient processing power to control the STB and run all 
native applications. There is a limited amount of processing power (the exact 
amount depends on what else needs to be done simultaneously) available to run 
other applications such as the l-frame grabber or EPG miner incorporated in the 
present invention. More advanced STBs feature higher performance processors 

15 and it is envisaged that the increased processing power provided by these 
advanced STBs may enable a wider range of applications to be implemented in 
the device of the invention and delays in providing images on the remote screen 
to be decreased. Desirably, the STB (or other gateway device) is provided with a 
second tuner which is separate from that which is used to select a channel for 

20 display on the TV display, the second tuner being controllable by the remote 
control device of the invention. STBs are known to be available which feature a 
second tuner for the purpose of recording a program other than the one being 
watched to a hard disk and it is expected that use could be made of this existing 
technology to enable many of the applications envisaged for the present 

25 invention. When a second tuner is present it is possible to access information 
(data or pictures) from any transport stream received by the TV receiver. 

The device of the invention may be connected to the STB (or other 
gateway device) by a serial cable but is preferably connected by a wireless link, 
such as a Bluetooth link. This can be achieved by attaching Bluetooth 'dongle' 

30 modules to the serial ports of the gateway devide and the remote control device. 
It is envisaged that existing, installed STBs and other gateway devices may be 
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upgraded in the field to provide functionality for the remote control device of the 
invention, by attaching such dongle modules and upgrading the flash memory of 
the gateway device with the relevant code. This provides a significant cost 
advantage to suppliers of gateway devices already in the field, who may provide 
a range of novel services to the consumer by means of a simple, low cost add-on 
to the existing generation of STBs and other gateway devices. 

An example of a suitable software framework for the invention is shown in 
Figure 2 below. The system requires a protocol for the transfer of commands 
and data between the remote control device and the gateway device (STB). 
The software stack used on the remote control device is based predominantly on 
freely available open source code. The software implements novel protocols to 
exchange information, interpret information and display the information using an 
appropriate GUI, for the device of the Invention. 

The system enables decoding of a second video stream in addition to a 
first that is decoded by the STB for display on the TV. The decoding of the 
second stream is conveniently carried out by means of suitably configured 
software on the main processor of a gateway device such as an STB. So as not 
to conflict with tasks already loaded on the processor and required by the STB or 
the cycle budget for these tasks (which can be unpredictable), the software 
decode desirably has a low processing load and is scalable so that it still worics to 
satisfaction even at times when the available cycle budget of the STB processor 
Is reduced. 

In a preferred embodiment, both temporal (throwing away frames) and 
spatial (reducing the resolution) scalability are used. In one example, only I- 
f rames are decoded and resolution is reduced eight times both horizontally and 
vertically. This can be implemented using the following method 

i) A PES filter for the 2nd data stream is set up in the demux of the 
decoder 

ii) Data is written into the PES buffer by the hardware demux 
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iii) If a full l-frame is found in the buffer, then decoding starts. (The 
decoding process is described in more detail below) 

iv) The decoded image is then sent to the display (or GUI) of the 
remote control device. Before display, the image format of the l-frames may be 

5 mapped onto a palette 

v) The PES filter is re-enabled and the cycle repeats. 

In one embodiment, l-frames are decoded and only the DC components of 
the 8x8 blocks are calculated. This gives a reduction in resolution by x 8. With 
this reduced resolution it is possible still to detect changes in image content (for 

10 example, whether a commercial break has ended and a soccer match resumed). 
A significant amount of processing load may be saved by using only DC 
coefficients, since this eliminates the need for performing IDCT operations. All 
OCT coefficients 8x8 blocks, (so also DCs) are coded in the incoming MPEG2 bit 
stream as VLC words. The DC coefficient is located at the start of each macro 

15 block. However macro blocks are not byte aligned. This means that in order to 
get to the DC of the next macro block, all AC coefficients of the previous macro 
block must be skipped. This implies that basically a full variable length decode 
operation must be performed, just like for full resolution decoding. This is costly 
bit-wise processing and consumes by far the largest part of the processing load. 

20 As only the length (rather than the exact value) of the AC coefficients are 

required, to skip AC coefficients, It Is possible to process the VLC coefficients 
more efficiently by using characteristics of the MPEG VLC tables. For example, 
analysis of bits for a particular coefficient is ceased once the length of the 
coefficient is known, and the remaining bits can be skipped. The desired result of 

25 the decoding is to end up with a progressive picture. This means that for macro 
blocks or DCT blocks that are field coded, field to frame conversion must be 
perfomied. This is can be achieved simply by averaging the DC components of 
the fields. 

The channel to be viewed In the display of the remote device may be 
30 changed without Interference with the image on the main TV display. When 
requested by the user the remote control device sends a channel change 
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command to the STB. The XML interpreter application handles this command by 
passing the channel number to the channel change function of the STB 
middleware. Many PDAs feature a built-in IR port which could also be used to 
control television functions such as volume, brightness etc if the correct remote 

5 control codes were known to the PDA. 

Other potential applications of the device of the invention not described in 
detail here include: remote teletext display of information from channels other 
than that viewed on the TV display; localisation of advertising and content; 
interactive and 'push' advertising; remote monitoring of programs viewed on the 

10 main TV display; immersive applications for sports programs; consumer loyalty 
schemes; universal front end/remote control; new user interfaces; EPG search 
engines; instant messaging on the remote display; alternate views for gaming 
consoles. 

The remote device 4 can receive information from a user via standard user 
15 interfaces such as keypad, touch screen including handwriting recognition and 
voice control. 

It is to be understood that the foregoing is merely exemplary of some 
embodiments and applications of the invention and is not intended to be limiting 
on the scope of the invention as defined in the appended claims. 
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